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I . INTRODUCTION 


During the 3 years that the Large Area Crop Inventory Experiment (LACIE) has 
been operational, it has become apparent that spring wheat must be distin- 
guished from other spring small grains such as barley, flax, and oats in 
order to obtain an accurate wheat area estimate. Because the spring small 
grain crops are so similar in their reflectance and development patterns, 
the analyst-interpreters have not been able to identify and label all of 
the separate spring small grain signatures with the same confidence with 
which they can label small grains as a group. Thus, the practice in LACIE 
has been simply to label and classify spring small grains as a class and, 
then, to generate a wheat estimate using a ratio of wheat to other spring 
small grains based on historical and economic variables. Such ratios are 
developed separately for each-stratum. 

The wheat ratios are not constant from year to year and are difficult to 
predict accurately. Therefore, a LACIE Transition Year goal is to reduce the 
dependence of the spring wheat estimates on the historical ratios of spring 
wheat to spring small grains and other confusion crops by estimating the 
proportion of spring wheat directly from the Landsat imagery. 

Prior to LACIE Phase III, a procedure was developed utilizing the ground- 
truth- data for 10 Phase II North Dakota blind sites. During Phase III, this 
procedure was used experimentally in the operational processing of the North 
Dakota sample segments. 

This plan describes the evaluation of the existing technology and the devel- 
opment of new and improved procedures for the direct estimation of spring 
wheat for application in the LACIE Transition Year. 
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2. APPROACH 


Three activities will be conducted to support the LAC IE initiation of the 
direct estimation of wheat in the LACIE Transition Year. First, the applica- 
tion of the direct wheat estimation procedure to North Dakota -in Phase III 
will be evaluated. Second, the direct wheat estimation procedure will be 
checked for transferability to the mixed grain states and to another year. 
Third, Research, Test, and Evaluation (RT&E) and Classification and Mensura- 
tion Subsystem (CAMS) design personnel will proceed with separate but coordi- 
nated research efforts in direct wheat estimation. An effort will also be 
made to determine the importance of 9-day coverage and to monitor the research 
by other organizations which relates to the direct wheat estimation procedure. 

2.1 EVALUATION OF THE PHASE III NORTH DAKOTA DIRECT WHEAT 
ESTIMATION PROCEDURE 

A direct wheat estimation procedure was used operationally in North Dakota 
during Phase III. (The procedure and the guidelines upon which it was based 
are presented in appendix A.) Ground-truth data were collected on 28 of the 
sample segments allocated within the state. Some of these 28 blind sites 
were never processed because of insufficient acquisitions, others were not 
able to pass the criteria for a satisfactory estimate, and still others 
had a very low percentage of small grains within the scene. ^ These factors 
combined to reduce the number of blind sites available for evaluation to 21. 

An additional three segments were deleted because Accuracy Assessment could 
not provide ground-truth tapes for segments with more than 500 fields. For 
each of the 18 remaining blind sites, summary statistics are being calculated 
and recorded. These statistics will be used to evaluate the procedure. 
Hopefully, they will be sufficient to answer the following questions: 

a. How accurately were the analysts able to label spring wheat dots? 

b. How accurately were the analysts able to label spring wheat dots which 
were classified as small grains? 

^The direct wheat estimation procedure applied in Phase III is not applicable 
to segments with a low density of small grains. 
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c. Are the accuracies obtained for discriminating wheat from other small 
grains by this procedure greater than those obtained from random chance? 

d. How accurately was the machine able to classify the 209 dots? 

e. The direct wheat estimation procedure assumes that the ratio of wheat to 
small grains is the same for those picture elements (pixels) classified 
as small grains as it is for those pixels classified as nonsmall grains. 

To what extent is this assumption tenable? 

f. How well do the LACIE wheat proportion estimates agree with the ground- 
truth wheat proportions? 

The forms upon which the data are to be recorded are presented in appendix B. 
The forms for the tabulations are in appendix C. These show in detail the 
terras to be quantified and the statistics to be generated. 

2.2 TRANSFERABILITY OF THE DIRECT WHEAT ESTIMATION PROCEDURE 

The direct wheat estimation procedure developed with North Dakota Phase II 
blind sites was tested on North Dakota Phase III blind sites and is proposed 

2 

for use on the LACIE Transition Year sites in the spring grain and mixed grain 
states of Minnesota, Montana, North Dakota, and South Dakota. Prior to the 
implementation of a direct wheat estimation procedure in LACIE operations for 
the Transition Year, two considerations for transferability will be addressed: 
(1) Is the procedure transferable to the other spring grain and mixed grain 
states? (2) Is the procedure transferable to another year? In order to 
answer these questions, the plots for the Phase III North Dakota direct wheat 
estimation procedure will be reproduced by examining data from the Phase III 
blind sites in the four spring grain and mixed grain states (appendix A). An 
intensive analysis of these plots will be conducted to determine which of the 
proposed characteristics of separation (guidelines) of the spring grains are 
consistent from year to year and state to state. Although somewhat subjective, 
these evaluations should provide some insight into the applicability of the 
procedure or a modified version of the procedure. 
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Mixed grain contains both spring and winter grain. 
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To begin this analysis, the collection of ground-truth data for the 209 dots 
must be completed for the blind sites in the other mixed grain states. Since 
the rate of flow of acceptable ground-truth data from Accuracy Assessment will 
not be sufficient to support this effort as scheduled, analysts are obtaining 
these data manually from ground-truth overlays of the aircraft photography. 

2.3 CAMS PROCEDURE DEVELOPMENT 

The improved direct wheat estimation procedure to be implemented in the 
Transition Year of LACIE will be developed in three basic parts. Part 1 will 
consist of scrutinizing the data collected and prepared for the Phase III 
evaluation (section 2.1) and the transferability study (section 2.2) to screen 
out all instances where separability is apparent. The different plots to be 
examined are presented in appendix D. In part 2, all the facts surrounding 
the instances of separability will be examined to determine when, under what 
conditions, and how consistently they occur. The results of parts 1 and 2 
will be conveyed to RT&E for interfacing with their research (section 2.4). 
During and after parts 1 and 2, the results from the RT&E activities will be 
reviewed. In part 3, all the research results will be collected together, 
and a search for a procedure which takes advantage of the newly found knowl- 
edge will be conducted. 

2.4 RT&E DIRECT WHEAT ESTIMATION RESEARCH 

In the direct wheat estimation research, RT&E will approach the problem 
using techniques and software that differ from those being used in CAMS. Time 
variables, raw data values, and Kauth-transformed data values essentially com- 
prise the CAMS developmental elements, whereas RT&E will utilize a software 
package which permits more flexible data manipulation. This effort will rely 
upon the data collection and results of the LACIE Phase III evaluation, the 
transferability study, and parts 1 and 2 of the CAMS procedure development 
activities. 

The potential benefit from the RT&E direct wheat estimation research will be a 
better presentation of the data than is currently available. In addition, the 



latest research from outside organizations will be monitored for its 
relevance to this activity and integrated when appropriate. 

2.4.1 ON-LINE PATTERN ANALYSIS AND RECOGNITION SYSTEM 

The primary tool to be used in this research is the On-Line Pattern Analysis 
and Recognition System (OLPARS), developed at the Rome Air Development Cen- 
ter, Rome, New York, and resident there on a Honeywell 6180 computer. The 
OLPARS is a highly interactive software package which combines a very 
flexible system for data storage, display, and manipulation with powerful 
techniques for data structure analysis and classification. In the context 
of this research problem, OLPARS will make possible the evaluation of the 
separability of wheat from other small grains in various one or two dimen- 
sional subspaces. These include: 

a. Coordinate vector projections 

b. Eigenvector projections 

c. Fisher's discriminant vector projections 

d. Generalized discriminant vector projections 

e. Any arbitrary vector projections, such as projections onto the 
brightness-greenness space 

In addition, OLPARS includes a routine for multidimensional scaling, which 
iteratively fits the data into a two or three dimensional subspace so that 
the difference between interpoint distances in the lower and higher dimen- 
sional spaces is as small as possible. Also included is the capability 
to perform any arbitrary linear transformation on the data before forming 
these projections. 

The structure analysis capabilities of the OLPARS will allow a quick and 
easy assessment of the separability of wheat from other small grains in 
a number of different two dimensional subspaces. This should provide a 
good evaluation of the separability of wheat from other small grains in 
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the original space. More importantly, it will greatly facilitate the 
development of two dimensional plots or projections to be used as analyst 
labeling aids. 


2.4.2 DATA SET 

The data set for this study will consist of ground-truth-labeled grid 
intersection points in each of several segments, including approximately 
20 segments from the 1976 crop year and approximately 25 segments from 
the 1977 crop year. For each segment the grid intersection pixels will 
be sorted by ground-truth class into wheat, barley, rye, oats, flax, and 
other. Each of these classes will be further divided into those pixels 
which are pure and those which are on a boundary. For example, there will 
be a pure wheat class and a boundary wheat class. Some of these classes 
may be combined later for analysis. 

The radiance values for all of the acquisitions available for each segment 
will be recorded; however, no segment will be chosen unless it has at least 
three acquisitions. 

( 

These data will be obtained from the CAMS data base on the Programmed Data 
Processor, model 11/45 (PDP 11/45). The data will be sent to the Rome Air 
Development Center, where they will be stored for access by OLPARS. 

2.4.3 ACQUISITION SELECTION FACTORS 

Because wheat and other small grains are so similar spectrally, the key to 
reliable discrimination is believed to lie in their temporal or developmental 
differences. Therefore, the selection of appropriate Landsat acquisitions 
Is very important. This study will examine the separability of wheat from 

other small grains as it relates to acquisition selection. 

2-4.3. 1 Separability as a Function of Crop Calendar Difference 

Because wheat is usually planted at a different time than other small grains 

and because different small grains have a somewhat different developmental 
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pattern, the separability of wheat from other small grains will be a function 
of time. The only keys generally available for assessing these developmental 
differences are the crop calendars for wheat and for other small grains. This 
phase of the study will attempt to relate the spectral separability of wheat 
from other small grains to the differences in these crop calendars. Separa- 
bility will be measured as the probability of correct classification (PCC) of 
ground-truth-labeled wheat dots using a standard OLPARS classification 
algorithm. The study will also make recommendations on the use of crop 
calendar information in selecting acquisitions likely to have the best 
separability of wheat from other small grains. 

The anticipated steps for accomplishing this phase of the study follow: 

a. Select the segments and acquisitions. 

b. Classify the grid intersection points for wheat versus small grains for 
each segment and acquisition, using the OLPARS maximum likelihood 
classification. 

c. Plot the PCC values for each acquisition of a segment versus the differ- 
ence in wheat and small grain crop calendars at the time of each 
acquisition. 

2.4. 3. 2 Mul ti temporal Separabi 1 i ty 

It is not clear how the separability of wheat from other small grains is 
affected by using multiple acquisitions. This part of the s.tudy will 
examine the separability of wheat from other small grains as a function 
of the number of acquisitions used. The acquisitions will be chosen 
using crop calendar information. An attempt will be made to quantify the . 
importance of acquisitions obtained at various times throughout the 
growing season. 

The anticipated steps for accomplishing this phase of the study follow. 
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a. Use the same data as in section 2.4.3. 

b. Classify the grid intersection points for wheat versus small grains, 
using two, three, and four acquisitions. (Use the QLPARS maximum 
likelihood classification.) 

t 

c. Plot the PCC values versus the number of acquisitions used for several 
segments. 

2.4.4 ANALYST AIDS 

In order for the analyst to be able to reliably discriminate wheat from 
other small grains, it is necessary that he have more information than 
simply the standard film products, trajectory plots, scatter plots, and 
other currently available ancillary information. This research will 
examine alternative analyst aids in the form of two dimensional subspace 
projections and will make recommendations concerning procedures for using 
these aids for the labeling of wheat. Two different approaches for the 
development of these aids will be used, one for fixed projections and one 
for segment-dependent projections. 

2.4.4. 1 Fixed Projections 

It may be possible to develop a fixed projection which can be used over a 
range of segments; for example, all of the segments in a stratum. Such a 
fixed projection would be particularly simple to use in operations and 
would be readily applicable to automated procedures such as Label Identifi- 
cation from Statistical Tabulation (LIST). 

The feasibility of a fixed projection will be explored by combining data 
from several segments and searching for a projection which provides suffi- 
cient separability of wheat from other small grains. If such a projection 
is found, it will be tried on segments in different areas and different 
crop years. The degree to which the projection for separating wheat from 
other small grains can be extended from segment to segment and from year 
to year will be evaluated. 
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2. 4. 4. 2 Segment- Depen dent Projections 

As an alternative to the fixed projection approach, a projection could 
be computed for each segment individually. This approach will be 
explored by searching for a type of projection that provides acceptable 
separability of wheat from other small grains when it is computed for 
each segment individually. 

The best such segment-dependent projection will be evaluated by determining 
which projection operation gives the best separability for a number of differ- 
ent test segments coming from different regions and different crop years. 

An important consideration will be whether the axes that result from segment- 
dependent projections are sufficiently stable to allow consistent rules for 
discrimination and interpretation. Ideally, the axes produced should 
have some physical interpretation. A minimum requirement is that fairly 
consistent rules for discriminating wheat from other small grains need to 
be developed. 

2. 4.4. 3 Development of Projections 

The steps to be used in accomplishing the fixed and segment-dependent 
projections are the following: 

a. Select 5 the segments. For the fixed projection problem,;- combine the 

, i 

data for several segments. 

b. Examine various two dimensional projections (i.e., Fisher's vector, 
eigenvector, generalized discriminate vector, and coordinate vector}. 

c. Select the candidate projections. 

d. Determine the PCC values when dots are labeled using the candidate 
projections. Use segments in different strata and different years. 

e. Compare the PCC values for the Fisher's and generalized discriminant 
vector projections {which should be optional in a discrimination sense 
but which require labeled samples) to the eigenvector and coordinate 
vector projections {which do not require labeled samples). 
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2.4.5 SUMMARY 

The goal of the research outlined in this plan will be to analyze acquisition 
selection and to develop analyst labeling aids. Both of these tasks will 
make use of the structure analysis and display capabilities of the OLPARS 
at the Rome Air Development Center. An overall consideration will be to 
develop techniques which are directly applicable from an operations stand- 
point. For this reason, the axes of the recommended projections will be 
as amenable to physical interpretation as possible. Also, procedures will be 
recommended for implementing acquisition selection techniques and analyst 
labeling aids for the discrimination of wheat from other small grains. 
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APPENDIX A 


WHEAT ESTIMATION GUIDELINES AND PROCEDURE 

The ability to separate wheat from other small grains will depend on crop 
development patterns. The spectral reflectance patterns of spring wheat and- 
other small grains are similar; however, general differences can be noted. 
These differences are: 

t Barley is generally planted after wheat. 

e Barley tends to green-up sooner than spring wheat and obtain higher levels. 
0 Barley turns and matures earlier than wheat. 

0 Barley tends to be brighter than wheat after heading. 

0 Rye is greener than wheat. 

0 Oats are not as green as wheat and may mature earlier than wheat. 

The green-number growth pattern for small grains is shown in figure A-l for 
general guidance. The assumption is that a field is wheat unless it can be 
shown that it is a nonwheat small grains. The production of wheat and other 
small grains in North Dakota is illustrated in figure A-2. Figure A-3 shows 
the process to be followed in deciding the wheat-small grains separation. 

The guidelines are designed to be flexible. 


^Personal communication with D. R. Thompson of the Applications Systems Veri- 
fication Branch of the National Aeronautics and Space Administration (NASA), 
June 1977. 


A-l 


Green number 



Date 


Figure A-1-— Green number growth pattern for small grains. 
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ALL WHEAT - 1975 

ALL WHEAT, 1975 - Production by Counties Percent Planted on Sumner Fallow 

(Rank of First Ten Counties Shown) (Percent) 



BARLEY, 1975 - Production by Counties 
(Rank of First Ten Counties Shown) 


OATS, 1975 - Production by Counties 
(Rank of First Ten Counties Shown) 



□ Under 250,000 bu. 

\ 250,000 - 499,999 bu. 


[ 500,000 - 2,499,999 bu. n Under 700,0 


U, 000, 000 - 1,499,999 bu. 


y$/\ 2,500,000 bu, & over 


700,000 - 999,999 bu. 1,500,000 bu. & over 


RYE, >1975 - Production by Counties 
(Rank of First Ten Counties Shown) 


FLAXSEED, 1975 - Production b> Counties 
(Rank of First Ten Counties Shown) 



| | Under 10,000 bu. jggggj 35,000 - 99,999 bu. □ Under 50,000 bu. 

j : -] 10,000 - 34,999 bu. ||||§| 100,000 bu. &.over CD 50,000 - 149,999 bu. ||||| | 


150.000 - 249,999 bu. 

250.000 bu. & over 


Figure A-2.— Production of wheat and other small grains in North Dakota. 
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Figure A-3.— Decision logic for separating wheat from other small grains 





A. 2 PROCEDURE FOR SEPARATING WHEAT FROM OTHER SMALL GRAINS* 


1. The bias corrected small grains estimate has been evaluated as 
satisfactory. 

2. Look at the 1975 county grain production maps to gain an understanding 

of the relative importance/ranking of each small grain in the county that 
the segment is located in. Use of the 9 x 9's at this time to check the 
relationship between the segment and county might prove helpful (see 
section 3.3 of the Phase III CAMS Detailed Analysis Procedures^ for 
amplification of use of Landsat full frames). 

3. Look at the crop calendar and the Small Grains-Wheat Separation Guidelines 
to formulate some expected general spectral characteristics (greenness and 
brightness) for each small grain. 

4. Look at the spectral plot of the base acquisition showing the classifier 
identified small grain and non-small grain pixels. Using the knowledge 
of the historical importance of each small grain in the county and the 
expected relative greenness and brightness position of each small grain, 
draw lines on the spectral plot to separate small grain classes. Label . 
these small grain classes on the spectral plot (i.e., wheat, barley, oats, 
etc.) within the boundary lines drawn. 

5. Locate each of the small grain pixels (using greenness and brightness 
values -from the spectral plot) in table 1, the ordered listing by dot 
number. 

6. For each pixel classified as an S (small grain) in the listing, indicate 
which grain class (wheat, barley, oats, etc.) previously indicated on the 
spectral plot that specific pixel belongs to. Put a symbol (W,B,0,R) to 
the right of the brightness value of that pixel in the table 1 base 
acquisition listing. 


*Personal communication with D. R. Thompson of NASA's Applications Systems 
Verification Branch and 0. D. O'Connell of Lockheed Electronics Company, Inc., 
June 1977. 

+ LACIE-00720, OSC-11693, August 1977. 
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7. Determine the total number of pixels classified as S in the classified 
column of table 1. 

8. Tally the number of S pixels in each grain class (W,B,0,R). 

9. Determine the proportion of classified small grain pixels for the 
wheat (W) class. 

Number of S pixels in the wheat class 
Total number pixels classified S 

10. Determine the percentage of spring wheat for the segment. 

(P^)(%SgQ^) = % of spring wheat 

where 

P w = proportion of SW 

SgcE ~ corrected estimate for grains 

11. Subtract the percentage of SW from the bias corrected estimate to obtain 
the percentage of other spring small grains in the segment. 

12. Record the proportions of SW and other SG on the CAMS Evaluation Form 
(CEF) and Porta-punch cards. 

13. Record all other information in the necessary places, put packet contents 
together in proper manner and turn packet over to the Spring Wheat/Small 
Grains Coordinator. 
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DATA TABLES 


HBLE B-l ACCURACY OF ALL 
MACHINE-CLASSIFIED DOTS 

Machine Segment: 

SG N Total Separation: 

SG 
N 

Total 








X 

Y 

<209 


TABLE B-2.- ACCURACY OF ALL 
DOTS MACHINE-CLASSIFIED 
AS NONSMALL GRAINS 


+-> 

13 

U 

4-> 


3 

O 

S- 

o 


SW 

B 

0 


Total 





Ground truth 


TABLE B-3.— ACCURACY OF METHOD FOR MACHINE-CLASSIFIED DOTS 
WITH SEPARATION LABEL 


Analyst/machine 



N, — Machine classified SG but 
analyst labeled NSG 


Statistics from printout : 

% ground truth data pure pixels 
% 209 pure pixels - 


Statistics from Accuracy 
Assessment (AA) printout: 


GT M/AI 


! P(1 » 
P( 2 , 
P(3 , 
P(4 , 
p(5 , 
P(6 , 

! P(* , 
P(* » 
P( * > 
p(,* » 

p(* , 

p(* , 

Key: 

1 - Wheat 
2 - Barley 
3 - Flax-rye 


'* ) 
* ) 
*) 
* ) 
* ) 
* ) 

1 ) 

2 ) 

3 ) 

4 ) 

5 ) 

6 ) 


4 — Oats 

5 -Alfalfa 

6 — N-other 
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TABLE B-4.— ACCURACY OF METHOD FOR 
ANALYST- LABELED DOTS 


Analyst 

SW B 0 N Total 


-C 

+-> 


s- 

+J 


c 

13 

o 

S-. 

CD 



TABLE B-5.— ACCURACY OF ANALYST-LABELED 
DOTS WHICH MACHINE CLASSIFIED AS OTHER 

. Analyst 
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APPENDIX C 
TABULATION TABLES 

TABLE C-1.- ACCURACY OF MACHINE CLASSIFICATION 

SEGMENT: 
SEPARATION 

1 . Pr (M = SG/GT = SG) = 

2. Pr (GT = SG/M = SG) = -jlj- 

3. Pr (correct classification) - 

(U + (4? 

(3) + (5) 

4. Pr (M = N/GT = N) = -j|j- 

TABLE C-2.— RATIO OF WHEAT TO OTHER SMALL GRAINS 

Machine 

N 

1. Actual ratio of SW to SG 

for dots classified N = 

(1) ; f 
(D + (2) + (3) ‘ . 

2. Pr (GT = N/M = N) = |||. 


SW 

B 

0 

N 

Total 


0 ) 


( 2 ) 


(3) 


(4) 


(5) 


AI GT 

TAPE GT [ZZ2 

Machine 
SG N Total 

SG 
N 

Total 


(1) 


(2) 


(4) 

(6) 

(3) 

X 

(5) 

Y 

5209 
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TABLE C-3.— ACCURACY OF METHOD FOR MACHINE-CLASSIFIED DOTS 
WITH SEPARATION LABEL 


Machine 

SW B 0 N^ G Total 
SW 
B 
0 
N 

Total 

N-, — Machine classified SG but 
analyst labeled NSG 


(1) 

(18) 


(15) 

(2) 


(3) 


(16) 

(4) 




(17) 

(6) 

(9) 

(10) 

(11) 

(7) 

(8) 

(12) 

(13) 

(14) 

(20) 

(21) 


SEGMENT: 

SEPARATION: 
GT TYPE: 


1 . 


Accuracies 

a. 

Spring wl 


Pr (AI = 


Pr (GT = 

b. 

Barley: 


Pr (AI - 


Pr (GT = 

c. 

Oats: 


Pr (AI = 


Pr (GT = 


SW/GT = SW & M 
SW/AI = SW & M 

B/GT = B & M = 

I 

B/AI = B & M = 

O/GT = 0 & M = 
0/AI - 0 & M = 


= SG) 

_ (U 
" T2T 

= SG) 

_ (1) 
" vm 

SG) = 

(3) 

W 

SG) = 

(3) 

TW 

SG) - 

(5) 

W) 

SG) = 

(5) 

TW 
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TABLE C-3.— Continued 


d. Nonwheat: 

Pr (AI = N/GT = N & M = SG) 



e. Small grains: 


Pr (AI = SG/GT = SG & M = SG) = 


(12) + (13) + (14) - (9) - (10) - (11) 
(21 + ' (4) + (6) 


•Pr (GT = SG/AI = SG & M = SG) = 

(2) + (4) + (6) - (15) - (16) - (17) 
(12) + (13) + (14) 


f. Ratios: 


Analyst estimate of ratio of SW to SG 

for dots classified SG = 

, 02 ) 

TT2) + (13) + (14) 

Actual ratio of SW to SG for dots 
classified SG = 

( 2 ) 

(2) + (4) + (6) 

g. Correct labeling overall: 

Pr (correct labeling overall) = 

(1) + (3) + (5) + (7) 

( 21 ) 
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TABLE C-3.— Concluded 


2. Confusions 

a. Barley: 

Pr (AI = B/GT = SW & M = SG) = 

Pr (GT = SW/AI = B & M = SG) = |j|| 

b. Oats: 

Pr {AI = 0/GT = SW & M = SG) = 

Pr (GT = SW/AI = 0 & M = SG) = 

c. Nonsmall grains: 

Pr (AI = N/GT = SW & M = SG) = -^|j- 

Pr (GT = SW/AI = N & M = SG) = |I|J- 

d. Spring wheat: 

Pr (AI = SW/GT / SW & M = SG) = 

(22) + (23) + (9) 

“(4) + (6) + (8) 
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TABLE C-4.— ACCURACY OF METHOD FOR ANALYST-LABELED DOTS 


AI GT 
TAPE GT ‘ 


SEGMENT: 

SEPARATION: 


Analyst 



SW 

B 

0 

N 

Total 

SW 

(1) 

(14) 

05) 

(16) 

(2) 

£ B 

u 


(4) 


(2) 

(5) 

-p 

~o 0 

c: 

3 ! 



(7) 

(21) 

(8) 

o 

u5 N 

(17) 

(18) 

09) 

(10.) 

(ID 

Total 

(3) 

(6) 

(9) 

(12) 

(13) 


1. Accuracies 

a. Spring wheat: 

Pr (AI = SW/GT = SW) = 

Pr (GT = SW/AI = SW) = 

b. Barley: 

Pr (AI = B/GT = B) = j|| 

Pr (GT = B/AI = B) = |H 

c. Oats: 

Pr (AI = O/GT = 0) = j|j- 
Pr (GT = O/AI = 0) = 

d. Nonwheat: 

Pr (AI = N/GT = N) = jjyj- 

Pr (GT = N/AI = N) = 
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TABLE C-4.— Concluded 


e. Overall accuracy: 

Overall accuracy = 

0) + (4) + (7) + (10) 

(13T 

f. Probability of correct labeling: 

Pr (correct label /GT = SG) = 

(1) + (4) + (7) 

(2) + (5) + (8) 

2. Confusions 

a. Barley: 

Pr (AI = B/GT = SW) = 

Pr (GT = SW/AI = B) = 

b. Oats: 

Pr (AI = O/GT = SW) = 

Pr (GT = SW/AI = 0) = -^||| 

c. Nonsraall grains: 

Pr (AI = N/GT = SW) = 

Pr (GT = SW/AI = N) = j]fj- 







d. 


Pr (AI = SG/GT = SG) = 

(3) + (6) + (9) - (17) - (18) - (19) 

(2) + (5) + (8) 

Pr (GT = SG/AI = SG) = 

(2) + (5) + (8) - (16) - (20) - (21) 

(3) + (6) + (9") 
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TABLE C-5 - ACCURACY OF ANALYST-LABELED DOTS WHICH 
MACHINE CLASSIFIED AS OTHER (N) 


Analyst 


13 

S- 

4-> 

a 

=5 

O 

S- 



SG 

N 

Total 

SG 

(1) 


(2) 

N 


(4) 

(5) 

Total 

(3) 


(6) 


SEGMENT: 

SEPARATION: 
GT TYPE: 


1 . Pr (AI = SG/GT = SG & M = N) = ||y 

2. Pr (GT = SG/AI = SG & M = N) = 

3. Pr (AI = N/GT = N & M = N) = 

4. Pr (correct label ing/M = N) = - - ^ /> ~ v ~ 


TABLE C-6.— OVERALL ACCURACY OF THE PROCEDURE 


1. Computation of ratios from the analyst- 
interpreted estimates: 

AT SW % 

Direct wheat ratio = ^ s~ q y 


GT ratio = 


GT SW % 
GT SG % 


2. Comparison of proportions for spring 
wheat and for small grains: 

AI SW % - GT SW % = ASW 

AI SG % - GT SG % = ASG 
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APPENDIX D 

GRAPHIC REPRESENTATION OF GROUND-TRUTH DATA FOR 
SPRING WHEAT, BARLEY, OATS, AND FLAX 



APPENDIX ft 


, GRAPHIC" REPRESENTATION OF GROUND-TRUTH DATA FOR 

ORIGINAL' PAGE IS SPRING WHEAT, BARLEY,, OATS, AND FLAX 

OF POOR QUALITY 

PLOT D-l - GREEN NUMBER VERSUS TIME 


SEG. 

NO. ACQUISITION DATES 



GREEN NO. 

VS TIME 


1699 77122 77140 7»l57 
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Green number ; Each space on the vertical axis represents a green number. 


Time : Time is expressed in 9-day increments, where 

SW — April 1 through September 30 

WW — September 1 through September 1 (omitting January and February) 

Columns : There are six columns per increment in the following order (left to 

right)': spring wheat (S), barley (B), oats (A), flax (F), winter wheat (W), 

and rye (R). 


Data representation : The data points are represented with digits reflecting 
the number of hits for that point. An asterisk (*) is used for all digits 
over 9. 
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PLOT D-2.— BRIGHTNESS VERSUS TIME 


_ BRIGHTNESS VS_ HH£ 

S»SP^ING WHETT 
BRIGHTNESS 

65 * ♦ *13 

* 1 


SEG. 

NO. 

. 132 . 9 . 

9c3A.SL£Y 


ACQUrSITI ON DATES 


~»*aATS” 


Ff FLAX 


_5 ft o n o 


W.WINTER WHEAT 
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1 _ 41 _ 

31 


"ST 

49 


2 

"ir 

13 

13" 

6 


11 * 

36 

42" 

25 


s — 
2 

221 
21 


13 

*4 


911 

11* 


55 ♦ 


2„ 
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TT 
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431 

67 

2A 

86 


TT 
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3 

7S 

34 




30 


a* 

43 

? 

11 


~rr 

? 

' 2i ’ 


SBAFUR' 

77091 


u 4 ♦ ♦ ♦ ; 

5euFH 3 S&itWsQA P dft SBAFkW SBaKUR SBA F Mft Sgir^B S3A F M H Ifli PWft CTIfW sBAPWft l*kt HR *B4FWR SBAFHft 


Brightness : Each space on the vertical axis represents a brightness value. 

Time : Time is expressed in 9-day increments, where 

SW - April 1 through September 30 

WW - September 1 through September 1 (omitting January and February) 

Data representation : The data points are represented with digits reflecting 

the number of hits for that point. An asterisk (*) is used for all digits 
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PLOT 0-3.- GREEN NUMBER VERSUS BRIGHTNESS 
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tftSHSls* S«slWJ** 'S{«)«X 


Note : This scatter plot is provided for each acquisition. 

Green number : Each space on the vertical axis represents one green number. 

Brightness : Each space on the horizontal axis represents one brightness value. 

Data representation : A data point is represented by the letter of the last hit. 

Listings : Two listings of data are provided for each plot, one ordered by 

green number and one ordered by brightness value. 
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PLOT D-3.— Continued 


(a) Listing ordered by green number 


SEG. ACQ. 
NO. DATE 
T640 77193 



“i*40 ?7l93 0" 0 “0 “0 “ 0“" 0* T * 0“ 'O'“*0 0 0 0 0 “fl "0 O'" 0 "ft - 6 
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'• PLOT D-3.— Concluded' 

(b) Listing ordered by brightness value 


SEG. ACQ. 

NO. DATE. 
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PLOT D-4.— GREEN NUMBER VERSUS ADJUSTED ROBERTSON SCALE 
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ACQUISITION DATES 
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Note : Separate graphs are plotted for spring wheat, barley, oats,.’;and flax 

ground-truth data. 

Green number : Each space on the vertical axis represents one green number. 

Robertson scale : Each space on the horizontal axis represents one-terith. 

Data representation : Data points are represented with digits reflecting the 

number of hits at that point. An asterisk (*) is used for all digits over 9. 
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PLOT D-5.— BRIGHTNESS VERSUS ADJUSTED ROBERTSON SCALE . 
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PLOT D-6.— NOMINAL CROP CALENDAR PLOTS 


(a) Green number versus nominal Robertson scale — crop plot for spring wheat 
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PLOT D-6.- Continued- 


. (b) Brightness versus nominal Robertson scale — crop plot for spring wheat 
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PLOT D-6.— Continued 


(c) Green number versus nominal Robertson scale — crop plot for barley 
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PLOT D-6.— Continued 


(d) Brightness versus nominal Robertson scale — crop plot for barley 
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PLOT D-6.“ Continued 


(e) Green number versus nominal Robertson scale - crop plot for oats 
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PLOT D-6.— Concluded 


(f) Brightness versus 

nominal 

Robertson 

scale 

— crop pi ot for oats 
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PLOT D-7.— RADIANCE VALUES FOR GROUND-TRUTH LABELS 
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PLOT D-7.— Continued 
(a) Multiple hits listing 
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PLOT D-7.— Continued 


(b) Mean and standard deviation listing 
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n3T Gflu NTS 

0,9 

0,0 

JU„. .. 

5,0 

3«7 

0,0 

0,0 

_ JL» 


7l’.7 5.4 

13*4 7,9 

B * _ 

67*. 8 
22* 4 


43.0 
5 , 0 


53,i {,4 

3,3 0,3 

2 *._. 


60*,6 9.1 

5,6 5*6 

-. 6 . 2 . 


9.6 

3,5 

6, 

67.9 

33.5 

1.6 

3,2 

3. 

64; 1 
26‘, 1 

7:7 
7*. 0 
62*. 

2*4 

4,9 

6 i 

-3?_*4 

2 .? 

3,6 

8,2 

2, 

43*. 7 
4,0 

4; 5 
3.4 
62’. 
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PLOT D-7.— Concluded 
(c) Median listing 


ME3|A% T48LE 


SEG. 

ACQ. 



BRIGHTNESS 




1604 

771*5 

CR0P 

NUMBER 
BF O0TS 

L8HEST 

VALUE 

23X 

VALUE 

MEDIAN 

value 

75X 

VAL'iE 

HIGHEST 
VAL *6 



W. WHEAT 

0 

0.0 

■n 

0.0 

o.o 

o;o 



BARLEY 

18 

25.9 


37,7 

49.8 

M.8 



RYE 

0 

oto— 

O', 0 

0.9 

0,0 

0.0 



HATS 

41 

3l'l 

4 2*. 7 

51.5 

59.3 ... 

85' 1 



FLAX 

6 

29.7 

29* 7 

*4.6 

*4.0 

- 75.4 



S. WHEAT 

AC 

26^5 

Zill 

11L6 

*6.6 

7C’.fl 


GREEN AUHdER 





NUMBER 

LOWEST 



75X 

H t GnE^T 

1604 

77125 

CR0P 

W. WHEAT 

0F D0TS 
3 

VAU'E 

c.o 

VAL»E 

0,0 

VALi-E 

o,o 

V \LUE 
0.0 

VALi«E 

c’.o 



barley 

RYE 

13 

0 

12.8 

0.0 

13.9 

o'.o 

14.5 

0.0 

15.1 

0.0 

mmm 



FATS 

FLAX 

41 

0 

10.2 
12' 7 

12.5 

12,7 

13W 

14.0 

■oa 

■nil 

n'.o 

14:5 



S, «REAT 

60 

U.2 

13,0 

13.9 

is Vo 

22.1 

■ 

BRIGHTNESS 

1604 

77143 

CR0P 

NUMBER 
0F D0TS 

lowest 

_.VA iL|iE_ 

25X 

_... VALUE 

MEDIAN 

VALUE 

75X 

VALUE 

HTGMEFT 

VAL^ic 



w, WHEAT 

barley 

0 

18 

5.6 

6C.0 

0.0 

65;3 

o.J 

68.8 

0.0 
7 3'. 1 

0.0 

75\9 



RYE 

**ATS 

3 

41 _ 

0.0 

92‘.0 

oto 

i7’,9 

0.0 

7l.| 

0.0 
76 » 2_ 

i 3.0 

SO' 6 



FLAX 

S, WHEAT 

6 

69 

64.0 
12x9 

64’. 0 
64’. 0 

66.3 

66x2 

66.3 
71,9 

iftz 

81' 3 


SREEN NUH9ER 



KUMBER 

L^EST 

25X 

MEDIaa 

75% 

HiGwtST 

1*34 77143 CR0P 

9F D0TS 


VALUE 

yIlue 

VlLUfc 

VaLi 6 

w. WHEAT 

0 

Ml 

O’.O 

0.3 

9.0 

O'.O 

BARLEY 

18 

mminm 

17 

20.3 

PI . 7 

36'. J 

RYE 

0 

rBlII 

0^0 

0.0 

0.0 

O'.O 

FATS 

41 


14.6 


19.2 

37.4 

FLAX 

6 

ir.i. 


13.5 

15.5 

17'. 5 

S, whEat” 

$"$ 

U.l 

i6.3 

17.4 

70. 6 

pKI 


ORIGINAL PAGE US 
OF POOR QUAtmg 
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